\documentclass[]{article}

\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{float}
\usepackage{xfrac}
\usepackage{marvosym}
\usepackage{ltxcmds} % get commands: \ltx@empty, \ltx@carzero
\usepackage{enumerate}
\usepackage{array}
%\usepackage[sort&compress]{natbib}

%\usepackage{hyperref}
\usepackage{amsfonts}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{textcomp}
\usepackage{amsthm}
\usepackage{bigstrut}
\usepackage[margin=1in]{geometry}
\usepackage{natbib}
\usepackage{graphicx}
\usepackage{listings}
\usepackage{color}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage{cancel}
\usepackage{chngcntr}
\usepackage{extarrows}
\usepackage{epstopdf}  
\usepackage{bigstrut} 
\usepackage{rotating} 
\usepackage{multirow} 
\usepackage{booktabs} 
\usepackage{authblk}

%dfdfd
\usepackage{epstopdf}
\epstopdfsetup{update}
\usepackage{epsfig}
\usepackage{booktabs}
\usepackage{pgf}
\usepackage{float}
\usepackage{subfig}
\usepackage{ifthen}
\usepackage{multirow}
%dfdfd



\theoremstyle{definition}% default plain
\newtheorem{thm}{Theorem}
\newtheorem{prop}{Proposition}
\newtheorem{cor}{Corollary}
\renewcommand{\qedsymbol}{$\blacksquare$}
\newtheorem{lem}{Lemma}
\newtheorem*{notation}{Notation}


\theoremstyle{definition}
\newtheorem*{defn}{Definition}
\newtheorem{conj}{Conjecture}
\newtheorem{exmp}{Example}
\newtheorem{assu}{Assumption}
\newtheorem{prope}{Property}
\newtheorem{note}{Note}
\newtheorem{remark}{Remark}
\newtheorem{obs}{Observation}
\newcommand{\Gbar}{\overline{G}}
\newcommand{\Hbar}{\overline{H}}
\newcommand{\Fbar}{\overline{F}}
\newcommand{\Ex}{\mathbf{\mathbb{E\/}}}

\newcommand{\pEx}[2]{\mathbf{E\/}\big(#1,#2\big)}
%\newcommand{\hr}[2] {
%\dfrac{#1(#2)}{\overline{\MakeUppercase#1}(#2)}}
%\newcommand{\hrt}[1] {
%\dfrac{#1(t)}{\overline{\MakeUppercase#1}(t)}}
%\newcommand{\Pr}{\mathbb{P\/}}
\newcommand{\diff}[2] {
\frac{d}{d #2}#1}
\newcommand{\hrate}[2] {
\mathbf{r}_{#1}(#2)}
\newcommand{\surv}[2]{\mathbf{\overline{F}}_{#1}(#2)}
\newcommand{\cdf}[1]{\mathbf{\overline{F}}_{#1}}
\newcommand{\pdf}[2]{\mathbf{{f}}_{#1}(#2)}
\newcommand{\MG}[2]{\mathbf{{MG}}_{#1}(#2)}
%\counterwithin{cor}{thm}
%\counterwithin{cor}{lem}
\newcommand{\noat}[1]{\textcolor{red}{ #1}}
\newcommand{\pr}{\mathbb{P}\/}
\newcommand{\ve}[1] {\mathbf{#1}}


\makeatletter
\newtheorem*{rep@theorem}{\rep@title}
\newcommand{\newreptheorem}[2]{%
\newenvironment{rep#1}[1]{%
 \def\rep@title{#2 \ref{##1}}%
 \begin{rep@theorem}}%
 {\end{rep@theorem}}}
\makeatother


\newtheorem{theorem}{Theorem}
\newreptheorem{theorem}{Theorem}
\newtheorem{corollary}{Corollary}
\newreptheorem{corollary}{Corollary}












\begin{document}
\bibliographystyle{authordate3}
\section{Modeling Framework}
We consider an ESRD patient receiving HD with at least one unused AVF opportunity. We assume that the patient does not consider other treatment options such as kidney transplantation and thus will depend on HD until death. Also, we assume that the patient has to choose between two vascular access types: CVC, and AVF; we do not consider AVGs here. We will discuss these limitations further in Section \ref{sec:dis}.


In Figure \ref{fig:decision}, the decision making framework is illustrated. As the decision flowchart suggests, we have made the following assumption:

\begin{assu} [Decision points] \label{ass:dec}
A patient can start the AVF creation process at any time, provided that an AVF is not under preparation or being used.
\end{assu}
Although it might be optimal to create a new AVF when the one being used is approaching the end of its lifetime, this is not clinically realistic, and thus we do not consider it here.

\begin{figure}[htbp]
\centering
\includegraphics[scale=.8]{./files/decision.pdf}
\caption{Vascular access choice for HD}
\label{fig:decision}
\end{figure}

The dynamics and principles of the model can be summarized as follows. A patient receives HD via an AVF as long as they have a functional one. When there is no functional AVF (whether when one fails or at the beginning of HD when the patient starts HD without an AVF) the patient dialyzes via a CVC as a bridge access. During this time, the policy in use determines \textit{\textbf{whether}} and \textit{\textbf{when}} to do the AVF surgery. If the policy in use recommends an AVF surgery, the patients goes through the AVF creation process, and wait until a functional AVF becomes available. The patient switches back and forth between CVC and AVF until when there is no more AVF opportunities, the patient dies, or when the policy in use does not suggest any more AVF surgeries.

We discuss factors impacting the decision of whether and when to use AVF opportunities in the following sections.
\subsection{Access-Based Patient Survival}
Patients survival on HD depends on the vascular access being used (\cite{Astor, KurellaSurv, perl_hemodialysis_2011}). Figure \ref{fig:survival_a}, obtained from \cite{perl_hemodialysis_2011}, shows that patients receiving HD via an AVF experience stochastically better survival than those who receive it via a CVC. Nevertheless, the survival benefit of AVF over CVC, measured by the failure rate difference, diminishes as patient continues using HD (see Figure \ref{fig:survival_b}, derived from Figure \ref{fig:survival_a}). In addition, patient's failure rate on either access types increases as the on-HD duration increases.

\begin{figure}[htbp]
  \centering
  \label{survival}
  \subfloat[Access-based survival probability.]{\label{fig:survival_a}\includegraphics[width=0.5\textwidth]{./files/surv_perl.pdf}}
  \subfloat[Access-based failure rate.]{\label{fig:survival_b}\includegraphics[width=0.5\textwidth]{./files/hrate_perl.pdf}}
\caption{Access-based survival probability and failure rate for a 67 year old on-HD patient. Figure \ref{fig:survival_a} is obtained from \cite{perl_hemodialysis_2011}, and Figure \ref{fig:survival_b} is derived from Figure \ref{fig:survival_a}.}
\end{figure}
We use these data-driven observations to justify further assumptions below. First, we describe some notation:

\begin{itemize}
\item $t$: time since the patient started HD
\item $\surv{X}{t}$: survival probability function of a random variable $X$ until time $t$ ($\surv{X}{t}=\pr [X > t]$) 
\item $\pdf{X}{t}$: probability density function of a random variable $X$ at time $t$
\item $\hrate{X}{t}$: hazard rate function of a random variable $X$ at time $t$ 
\item $X_t$: residual lifetime of a random variable $X$ at time $t$ (a random variable denoting the remaining lifetime of $X$ from time $t$ onward conditional on survival until time $t$)
\item $\mu(t) \in \{{a, c}\}$: patient's HD modality at time $t$ ($a$ if it is an AVF, and $c$, if it is a CVC).
\item $C$: random variable denoting patient's lifetime if they were to remain on a CVC from HD initiation time until death.
\item $A$: random variable denoting patient's lifetime if they were to remain on an AVF from HD initiation time until death.
\item $L$: random variable denoting patient's lifetime
\end{itemize}
Note that the distributions of $C$ and $A$ are dependent on a patient's age at the time HD commences, but we do not denote this dependency for ease of notation.

The followings are the definitions for two common types of stochastic order for random variables.
\begin{defn} [Usual stochastic order]
We say $X \le_{st} Y$, if and only if
\begin{align*} %\label{eq:stdef}
\surv{X}{t} \le \surv{Y}{t}: \forall t.
\end{align*}
\end{defn}
\begin{defn} [Hazard rate order]
We say $X \le_{hr} Y$, if and only if
\begin{align}\label{eq:hrdef}
\hrate{Y}{t} \le \hrate{X}{t}: \forall t.
\end{align}
\end{defn}

 Our first assumption describes how survival depends on HD duration and vascular access type.
\begin{assu} [Survival distribution] \label{ass:surv}
A patient's survival depends on the length of time that the patient has been on HD and the current mode of HD access (an AVF or a CVC), and is independent of the history of HD access type.\\
Mathematically, Assumption \ref{ass:surv} indicate that if the patient remains on the same access from $t$ until $t + x$ (for any $x \ge 0$), we have:
\begin{align}
& \pr \left(L_t \ge x \big| \mu(t') :\forall t' \le t\right)=\pr \left(L_t \ge x \big| \mu(t)\right), \label{eq:markovian} \\
&\pr \left ( L_t \ge x \big| \mu(s)={a} : t \le s \le x+t\right) =\surv{A_t}{x}, \label{eq:sAVF}\\
&\pr  \left (L_t \ge x \big| \mu(s)={c} : t \le s \le x+t\right)=\surv{C_t}{x}. \label{eq:sCVC}
\end{align}
For technical purposes, we also assume that $\surv{A}{x}$, and $\surv{C}{x}$ are differentiable at all values of $x$. Note that this implies differentiability for  $\surv{A_t}{x}$, and $\surv{C_t}{x}$, as well.
\end{assu}

Equation \ref{eq:markovian} states that a patient's survival depends only on current access type and is independent of her HD access history. Equations \ref{eq:sAVF} and \ref{eq:sCVC} indicate that a patient who has survived until $t$ and whose HD access would be the same from $t$ until at least $t+x$ experience the same survival as if she has had the same access type from HD initiation until $t+x$ and has survived until $t$.

The Markovian property of the survival (independence of future survival from access type history) has been made in Assumption \ref{ass:surv} for the sake of modeling tractability, and also because there is no data-driven support for a finer consideration. Our discussions with our clinical collaborator suggested that a finer modeling of history would be a second order effect for developing model-based insights. 

The following three assumptions formalize our observations from Figure \ref{fig:survival_b}.
\begin{assu} [Relative performance]  \label{ass:relative}
The residual lifetime of $C$ is stochastically smaller than the residual lifetime of $A$, at all ages. Mathematically, we have:
\begin{align*} 
 C_t \le_{st} A_t , \forall t
\end{align*}

Note that Assumption \ref{ass:relative} is equivalent to $ C \le_{hr} A$  (see Lemma \ref{lem:hr_eq} of the Appendix xx) which corresponds to the CVC hazard rate curve lying above the AVF hazard rate curve in Figure \ref{fig:survival_b}.
\end{assu}
\begin{assu} [Diminishing difference] \label{ass:converging}
The difference between hazard rates of $C$ and $A$ decreases in time, i.e. 
\begin{align*} %\label{eq:conv}
\hrate{C}{t} - \hrate{A}{t} \downarrow t.
\end{align*}
\end{assu}
Note that Assumption 4 corresponds to the diminishing gap between the CVC hazard rate curve and the AVF hazard rate curve of Figure \ref{fig:survival_b}.

Finally, the following assumption states that the performance of HD on either access types, measured by the failure rate, diminishes as the patient ages.
\begin{assu} [Diminishing performance] \label{ass:IFR}
Random variable $A$ and $C$ have the increasing failure rate (IFR) property, i.e. $\hrate{A}{t}$ and $\hrate{C}{t}$ are increasing in $t$.
\end{assu}
Assumption 5 is demonstrated by the fact that both curves in Figure \ref{fig:survival_b} are increasing.  

\subsection{AVF Creation Process}
After a patient and her clinician decide to use an AVF for HD, she visits a vascular surgeon for AVF placement and waits for an AVF surgery time. After the surgery is performed, the AVF maturation, a process by which a fistula becomes suitable to use for HD (e.g. develops adequate flow, wall thickness, and diameter) begins. The time from AVF creation to achieve an AVF usable for HD (with interventions if necessary) or AVF abandonment due to failure takes about 3 months (\cite{rayner,Ethier}). Around 60\% of placed AVFs fail to mature (\cite{Dember2, Dember,Hakim, Xue}). Furthermore, even if an AVF creation is successful, it has a limited lifetime (\cite{Roy, Radoui}).  These factors are critical to the optimal timing of AVF referral. 

We use the following notation for random variables describing the AVF creation process:
\begin{itemize}
\item $W_i$: random variable denoting wait time from AVF referral until AVF surgery of the $i$th AVF creation
\item $M_i$: random variable denoting the maturation time of the $i$th AVF
\item $K_i$: random variable denoting the total lifetime of the $i$th AVF  (if AVF creation is unsuccessful, then $K_i=0$)
\end{itemize}
We make the following assumption about the AVF creation process:
\begin{assu} [AVF's maturation and lifetime] \label{ass:AVFs}
All respective random variables describing the AVF creation process ($W_i, K_i,M_i$) are identically and independently distributed. Furthermore, their distributions are stationary and are independent of the survival process.
\end{assu}
Note that these conditions can be relaxed to accommodate the negative impact of aging on the AVF creation process variables. For instance, the results still hold when the AVF maturation is (stochastically) longer, and/or if the AVF lifetime is (stochastically) shorter for the AVFs that are created at a later time. Also, note that HD dependent patients have a rather short lifetime expectancy and thus assuming stationary distributions seems reasonable. For the sake of model tractability and ease of exposition, we make this assumption. 

\subsection{Objective Functions}
\subsubsection{Total Lifetime}
A natural metric for comparing policies is the total lifetime of a patient. Thus, we consider maximizing a patient's total lifetime as one of the objective functions. 

\subsubsection{Quality Adjusted Life Expectancy (QALE)} \label{sec:QALEDef}
Using AVF for HD not only brings better survival, but also has a slightly higher quality of life for the patient, in comparison with HD using a CVC (\cite{Goro, Lopez}). Nevertheless, the process of AVF creation has some disutility associated with it, which can be attributed to the surgery and post-surgery inconveniences, complications or costs. We define a patient's quality adjusted life expectancy (QALE) as the quality adjusted lifetime on each vascular access minus the AVF surgery disutility for each AVF surgery performed (whether successful or not).

The following parameters are used in defining the patient's QALE:
\begin{itemize}
\item $L_A^{\pi}$, $L_C^{\pi}$: random variables denoting patient's aggregate lifetime on AVF and CVC, respectively, from HD initiation until death, under an arbitrary AVF referral policy $\pi$
\item $q_a$, $q_c$: quality of life coefficient of HD using AVF and CVC, respectively
\item $d$: AVF creation disutility
\item $N^{\pi}$: random variable denoting number of AVF surgeries performed, under an AVF referral policy $\pi$
\item $Q^{\pi}$: random variable denoting the patient's total QALE, under an AVF referral policy $\pi$
\end{itemize}

Using this notation, we have the following equation for a patient's total QALE under an arbitrary AVF referral policy $\pi$:
\begin{align*}
Q^{\pi}=q_a L_A^{\pi}+q_c L_C^{\pi}-dN^{\pi}
\end{align*}
Based on the estimates in the literature (see \cite{Goro,Lopez}), we make the following assumption about the access-based quality of life coefficients.
\begin{assu}[Relative quality of life]\label{ass:qol} Patients experience a better quality of life dialyzing via an AVF than via a CVC; Mathematically, we assume $q_a \ge q_c$.
\end{assu}

\section{Structural Properties}
In this section, we ...
\subsection{Optimal Policy: Total Lifetime} \label{sec:optTL}
Since the survival benefit of an AVF over a CVC decreases as the patient ages, one may think an HD patient should be referred for AVF as soon as an opportunity becomes available, rather than keeping the opportunity for later years.  We prove this below in a stochastic ordering sense: an identical patient referred for AVF earlier than another patient lives stochastically longer than that patient. Of course, this also means that the first patient has a longer expected lifetime.

\begin{thm} \label{thm:total}
Under Assumptions \ref{ass:dec}-\ref{ass:converging}, and \ref{ass:AVFs}, delaying AVF referral stochastically decreases a patient's lifetime.
\end{thm}
Note that this theorem implies that the optimal policy to maximize the probability of survival until any time $t \ge 0$ (and as a result to maximize expected lifetime) is to refer a patient on CVC for an AVF creation as soon as possible, provided AVF opportunities remain and no AVF is already in the process of maturing.

\subsection{Optimal Policy: QALE} \label{sec:QALEGP}
Since the survival benefit of AVF decreases as a patient ages, one would want to avoid the AVF creation disutility if it cannot be compensated by the better survival and quality of life associated with HD on an AVF. We will show that the optimal referral policy to maximize a patient's expected QALE is of threshold type: if the patient's age at the time of decision is less than a critical age, i.e. if $t<\tau^*$, then it is optimal to refer patient for AVF creation at the time of decision; otherwise, the optimal policy is to use CVC for the rest of patient's life.  We will also prove that the critical age is independent of the number of AVF chances remaining.

To explain the dynamics of the model and prove the results, we formalize the decision making process with a dynamic programming model using the following notation (see Figure \ref{fig:dp} for an illustration of the model).
\begin{itemize}
\item $\pi(\tau)$: a threshold policy with parameter $\tau$ 
\item $\pi_0$: the policy of using CVC for the rest of the patient's life (hereafter referred to as the no-referral policy). Note that $\pi_0 $ is equivalent to $\pi(\tau)$ for $\tau=0$.
\item $v^\pi(t,n)$: the value function (the residual QALE of a patient) at decision state $(t,n)$ under policy $\pi$
\item $v(t,n)$: the optimal value function at decision state $(t,n)$.
\item $@_y$: the decision to refers patient for AVF creation at time $y$ (with respect to the time of decision)
\item $v(t,n,y)$: the value function of the policy consisting of action $@_y$ for the current AVF chance, and then following the optimal policy for the subsequent decisions
\item $L(t,n,y)$: random variable denoting patient's residual lifetime at time $t$ under the policy mentioned above for $v(t,n,y)$
\end{itemize}

\begin{figure}[htbp]
\centering
\includegraphics[scale=0.6]{./files/dp.pdf}
\caption{Dynamic programming diagram. The diagram shows the dynamics of decisions made about possible AVF surgeries for a patient, at time $t$. It shows a scenario in which, for the current AVF chance, a surgery is planned at time $t+y$, and depending on patient's survival (determined by the random variable $L(t,n,y)$), a future decision for the subsequent AVF chance is due at time $t'$, which itself is determined by random variables $M$ and $K$.}
\label{fig:dp}
\end{figure}

The elements of the dynamic programming model are as follows:
\begin{itemize}
\item \textbf{States}: The set $\big \{(t,n), \forall t,n\ge 0 \big\} \cup \big\{\Delta\big\}$ defines the state space. The set of vectors $(t,n)$ consisting of $t$, the time, and $n$ the number of AVF chances left, corresponds to a living state, and the absorbing state $\Delta$ corresponds to the death state. A decision state is a living state in which we have $n \ge 1$.
\item \textbf{Actions}: In each decision point $(t,n \ge 1) $, one of two actions can be taken: either to refer patient for AVF creation at time $y$, denoted by $@_y$, or no more AVF referrals (the no-referral action). Note that the  no-referral action is the case of referral at $y=\infty$. Nevertheless, we keep it in the action space for  clarity. When $n=0$, the only option is to remain on CVC for the remainder of the patient's lifetime.  
\item \textbf{Transition probabilities}: We may transition to the state $\Delta$ or to $(t',n-1)$ for some $t' \ge t$. More specifically, we transition to the state $(t'=t+y+M_n+K_n,n-1)$ if $L(t,n,y) \ge y+M_n+K_n$, and to the state $\Delta$, otherwise. Note that the residual lifetime, $L(t,n,y)$, is dependent on $M_n,K_n$ as well.

\item \textbf{Immediate reward}: The immediate reward consists of patient's total QALE from time $t$ to the next state. Define the reward function $r(y,m,k,l)$ as follows, in which $m,k$ and $l$ are instances of random variables $M,K$, and $L(t,n,y)$, respectively (see Figure \ref{fig:dp}):
\begin{align*}
r(y,m,k,l)=\begin{cases}
q_cl& l \le y\\
-d+q_c l& y \le l \le y+m\\
-d+q_c(y+m)+q_a\big(l-(y+m)\big) & y+m \le l \le y+m+k\\
-d+q_c(y+m)+q_ak & y+m+k \le l
\end{cases}
\end{align*}
Then, the immediate reward of action $@_y$ is defined as $R(t,@_y):=\Ex r\big (y,M,K,L(t,n,y)\big )$.
\item \textbf{Optimality equation}: The Bellman optimality equation is as follows:
\begin{align*}
v(t,n)= \max \{ \sup_y v(t,n,y), v^{\pi_0}(t,n) \}
\end{align*}
in which 
\begin{align*}
& v(t,0)=v^{\pi_0}(t,n)=R(t,@_\infty)=q_c\Ex C_t,\\
& v(t,n,y)= R(t,@_y)+\pr[L(t,n,y) \ge t'(y)-t].\Ex_{t'(y) \big | L(t,n,y) \ge t'(y)-t} \big[ v(t',n-1) \big].
\end{align*}
\end{itemize}
The following proposition proves the optimality of threshold policies for the case in which we have one AVF opportunity remaining ($n=1$).
\begin{prop}[Existence of a referral threshold for $n=1$] \label{prop:qalen=1}
Assume $n=1$. Under Assumptions \ref{ass:dec}-\ref{ass:qol}, there is an age threshold $\tau^*$ such that the policy $\pi(\tau^*)$ maximizes the expected QALE of the patient. In other words, for $t < \tau^*$, referral at $t$ is the optimal action, otherwise, the no-referral action is  optimal.
\end{prop}
Note that for simplicity of notation, we don't show the dependency of the threshold on the age at HD initiation and other model parameters. 
\begin{cor}[Binary search]\label{cor:binsearch} The optimal policy can be found using a binary search for $\tau^*$ over $[0,\infty)$.
\end{cor}
In the next theorem, we see that the optimal threshold found in Proposition \ref{prop:qalen=1} for $n=1$ is also optimal for $n > 1$. Therefore, it establishes the fact that the decision of whether and when to refer for AVF creation is independent of the number of AVF chances remaining.
\begin{thm} [Optimality of threshold policies] \label{thm:QALE}
The policy $\pi(\tau^*)$ (constructed in Proposition \ref{prop:qalen=1}) is optimal for all $n \ge 1$.
\end{thm}


\subsection{Critical disutility}\label{sec:critdis}
In Section \ref{sec:QALEGP}, we proved that there exist an age-threshold policy that maximizes a patient's QALE. The AVF creation disutility, one of the factors determining the optimal policy, is the hardest parameter to estimate, and there is also a considerable heterogeneity among patients in that regard. To circumvent this, we introduce a dual view of the age-threshold policy. We show that at any time, the decision of whether to do an AVF surgery or not is determined by comparing the patient's AVF creation disutility with a critical value. Thus, in order to make a decision, we only need to know whether the AVF creation disutility is above the critical value or not, and do not require to know the value itself. This will also facilitates nephrologists getting patients involved in the AVF referral decision making process.

\begin{thm} [Critical disutility] \label{thm:cdis} At any age, there exist a non-negative critical AVF creation disutility, denoted by $d^{\text{cr}}(t)$, such that the optimal decision at time $t$ is to do an AVF surgery immediately if the patient's AVF creation disutility is less than the critical disutility (i.e. if $d \le d^{\text{cr}}(t)$), and is to use CVC for the rest of patient's life, otherwise.
\end{thm}

As explained in the proof of Theorem \ref{thm:cdis}, the critical disutility at time $t$ is the difference of the expected quality adjusted lifetime between a scenario in which a patient performs AVF creation surgery at time $t$ and benefits from her (possible) AVF access after the maturation time is over, and the scenario in which the same patient uses a CVC as the HD access from time $t$ until death. In other words, it is the gross QALE benefit of having one AVF chance (and doing the AVF surgery immediately) over depending solely on a CVC, without subtracting the AVF surgery disutility. We use this interpretation in explaining the sensitivity of the optimal policy to the model parameters.


Based on Theorem \ref{thm:cdis}, we have the following immediate result:
\begin{cor}\label{cor:dec_d}
The critical age, $\tau^*(d)$, is decreasing in $d$.
\end{cor}

We also have the following intuitive result for $d^{\text{cr}}(t)$:
\begin{cor}\label{cor:dcrt}
Critical disutility, $d^{\text{cr}}(t)$, is decreasing in $t$. Furthermore, if any of Assumptions \ref{ass:relative}-\ref{ass:IFR} hold strictly, we have that $d^{\text{cr}}(t)$ is strictly decreasing in $t$.
\end{cor}
Since critical disutility is decreasing in $t$, in the case that a patient's AVF creation disutility is non-decreasing in time, the optimal policy is still a threshold policy, $\pi(\tau^*)$, in which $\tau^*$ is the smallest $t$ where $d(t) > d^{cr}(t)$.



In the following corollary, we show that if there is no disutility for AVF creation, then at all ages the optimal policy to maximize the expected QALE of a patient is to refer patients for AVF as soon as an opportunity becomes available. Note that assuming $q_a=q_c=1$ and $d=0$, we have that QALE equals the lifetime of the patient. Thus by Theorem \ref{cor:d=0}, the optimal policy to maximize the expected lifetime of the patient is to refer immediately at all ages. This coincides with the result of Theorem \ref{thm:total}. Nevertheless, Theorem \ref{thm:total} gives a stronger result than Corollary \ref{cor:d=0}, because it not only gives an ordering of actions, but also gives a strong ordering (the stochastic order), whereas Corollary \ref{cor:d=0} only gives the optimal action, and also under a weaker order (expected value order).

\begin{cor} \label{cor:d=0}
If $d=0$, then $\tau^*=\infty$.
\end{cor}

 

Theorem \ref{thm:cdis} also provides an alternative way of comparing the optimal policy for individual patients as follows: if the critical disutility for one patient is always smaller than another, then the first patient has a smaller age-threshold, given that both patients have the same AVF creation disutility.

Recall that a patient characteristics can be summarized by the quality of life factors ($q_a, q_c, d$), distributions of random variables defining patients survival ($A$ and $C$, which depend on the patient's age at HD onset), and random variables describing the AVF creation process and lifetime ($K$, $M$, and $W$). Thus, we perform comparative statics to show how the optimal policy changes according to an individual's preferences and/or physical characteristics.  

\begin{thm}\label{thm:compdcrt}
Consider two patients types indexed by 1 and 2 whose characteristics satisfy the following properties:
\begin{enumerate}
\item $M^{(2)} \le_{st} M^{(1)}$,
\item $K^{(1)} \le_{st} K^{(2)}$,
\item $q_a^{(1)} \le q_a^{(2)}$,
\item  $q_a^{(1)} - q_c^{(1)} \le q_a^{(2) }- q_c^{(2)}$,
\item $A^{(1)} \le_{hr} A^{(2)}$,
\item $[\hrate{C^{(1)}}{t}- \hrate{A^{(1)}}{t}] \le [\hrate{C^{(2)}}{t} -\hrate{A^{(2)}}{t}]: \forall t$,
\end{enumerate}
where $(i)$ denotes the patient's index. Then, $d^{\text{cr}}_{(1)}(t) \le d^{\text{cr}}_{(2)}(t): \forall t$.
\end{thm}


This theorem states conditions under which the gross benefit of having an AVF chance is higher; In terms of the AVF creation process, the critical disutility is higher when the AVF maturation time is (stochastically) shorter, and/or if the AVF lifetime is (stochastically) longer. With respect to the QALE parameters, the gross benefit of an AVF is higher when an HD-dependent patient has a better life quality factor (whether via a CVC or an AVF), and/or when the difference of life quality factor between an AVF and a CVC is higher. A similar argument applies to the patient's survival, i.e. when a patient has a better HD-based survival and/or a better AVF-based on-HD survival (compared with a CVC-based survival), then the critical disutility is higher.

We continue this section by discussing a special case of this problem.
\subsection{A Special Case}
In this section, we discuss a special case in which we assume that a patient's lifetime on HD follows exponential distribution. The importance of this special case is three fold:
\begin{enumerate}
\item We can find a closed form for the critical disutility. Using the closed form, we can demonstrate how different components of the model interact.
\item It provides an upper-bound on the critical disutility.
\item Exponential distribution for survival has been considered in other related research works (see \cite{Xue}). Furthermore, exponential distribution is a plausible assumption when the patient has a short life expectancy.
\end{enumerate}

Assume $A \sim \exp(a)$, and $C \sim \exp(c)$. Note that $\hrate{A}{t}=a$ and $\hrate{C}{t}=c$, and therefore Assumptions \ref{ass:relative}-\ref{ass:IFR} are satisfied if and only if $a \le c$. Therefore, we assume $0<a \le c$ for the rest of this section.

The optimal policy for the total lifetime metric is independent of the distributions of $A$ and $C$. Therefore, we only discuss the optimal policy for the QALE metric. As Theorem \ref{thm:QALE} suggest, there is an age-threshold when the optimal policy switches from immediate AVF surgery  to use CVC forever. But since the exponential distribution has the memoryless property and lacks any notion of aging, the optimal decision must be the same at all ages. We prove that implicitly as follows:

\begin{thm}\label{thm:exp}
Assume that $A \sim \exp(a)$, and $C \sim \exp(c)$, where $0<a \le c$. Then,
\begin{align}\label{eq:components}
d^{\text{cr}}(t) =d^{cr}:= p\bigg[\frac{q_a}{a}-\frac{q_c}{c} \bigg] \bigg[\Ex_M\big[ e^{-cM}\big]\bigg ] \bigg[1-\Ex_Z\big[ e^{-aZ}\big] \bigg].
\end{align}
where $p$ is the success probability of the AVF creation process, and $Z$ is the lifetime of a functional AVF. Furthermore, if $M\sim \exp(m)$, and $K\sim \exp(k)$, we have:
\begin{align*}
d^{\text{cr}} = p .\bigg[\frac{q_a}{a}-\frac{q_c}{c}\bigg] .  \frac{1}{1+ \frac{m}{c}} . \frac{1}{1+ \frac{a}{k}} 
\end{align*}
\end{thm}
Since $d^{\text{cr}}$ is a constant term (with respect to $t$), the optimal decision is to always (i.e. at all ages) refer immediately, if $d \le d^{\text{cr}}$, and never refer at all, otherwise.

Equation \ref{eq:components} also demonstrates how different components of the model interact. The critical disutility is product of four parts:
\begin{enumerate}
\item $\dfrac{q_a}{a}-\dfrac{q_c}{c}$: this represents the ideal QALE benefit of an AVF over a CVC, which we could achieve if a) AVF could mature right away, b) the surgery would always be successful, and c) AVF lived forever.
\item $0 \le \Ex_M[ e^{-cM}] \le 1$: this factor represents the impact of the maturation time. The ideal QALE benefit is downscaled by this factor because the benefit applies only when the maturation time is over and the patient switches to the prepared AVF. Until this time, the patient have to use a CVC to receive HD.
\item $p$: this represents the impact of success probability of AVF creation. The patient benefits from using an AVF only when the AVF creation is successful.
\item $0 \le 1-\Ex_Z [ e^{-aZ}] \le 1$: this factor represents the impact of (limited) lifetime of an AVF. Even if the patient survives until she switches to a successfully matured AVF, the lifetime of an AVF is limited and she has to switch back to a CVC when the AVF fails.
\end{enumerate}
Note that this result is in agreement with results of Theorem \ref{thm:compdcrt}. For instance, the critical disutility defined in Equation \ref{eq:components} is higher when AVF maturation time, $M$, is (stochastically) smaller.

It is worth mentioning that for this case the critical disutility can be easily calculated (recall that $\Ex_X[e^{-tX}]$ is the moment generating function of a random variable $X$ evaluated at $-t$).\\

In the following corollary, by combining the result of Theorems \ref{thm:compdcrt} and \ref{thm:exp}, we provide an upper-bound for the critical disutility.
\begin{cor} The following holds for the critical disutility:
\begin{align*}
d^{\text{cr}}(t) \le p\bigg[\frac{q_a}{\hrate{A}{t}}-\frac{q_c}{\hrate{C}{t}}\bigg]\bigg[\Ex_M\big[ e^{-M \hrate{C}{t}}\big]\bigg ]\bigg[1-\Ex_B\big[ e^{-B\hrate{A}{t}}\big] \bigg]
\end{align*}
\end{cor}
The upper-bound can be used when the survival data is censored. We discuss this further in the next section.

For the especial case, we can observe that the critical disutility is linear in the AVF creation success probability. In the following theorem, we show that this is a general result:
\begin{thm}\label{thm:pavf} Critical disutility is linear in the AVF primary failure rate.
\end{thm}
Based on this theorem, the critical disutility can be easily adjusted by the nephrologist based on their perception of a patient's AVF primary failure rate.

\subsection{Censored survival tables}
In most of clinical studies in which patients survival are evaluated or compared under different scenarios (e.g. random trials studying different treatment options), patients are rarely followed up until their death, and thus the survival data suffers from right censoring. As a result, it is often needed to extrapolates the existing data when the effects of the decisions made go beyond the time horizon of survival studies (\cite{economic}). 

Patient survival on HD is not an exception to censored data; for instance \cite{Astor}, \cite{KurellaSurv}, and \cite{perl_hemodialysis_2011} have reported survival probabilities up to 5 years after HD initiation. Thus, we also need to extrapolate survival data in this study in order to numerically compute the parameters of the optimal policy. 

There are different ways and options in conducting extrapolations, and they may result in contradicting recommendations. Therefore, much attention has been paid to the issue of extrapolating patient's survival recently, especially in the cost-effectiveness analyses (e.g. see \cite{EditExtrap,MainExtrap,hazards}). 

When comparing different treatment options, the critical question is what to assume about the relative performance of these alternatives in the unobserved period. Grieve et al. discuss the issue of survival extrapolation when studying a new treatment and recommends considering the following four scenarios about the treatment effect in the unobserved period when extrapolating the survival data (\cite{EditExtrap}): 1- no treatment effect in the unobserved period 2- a treatment effect that declines over time, 3- a continuation of the main treatment effect, and 4- to predict treatment effects over time in the unobserved period using observed information (the first three scenarios are based on  the 2013 National Institute for Health and Care Excellence technology appraisals guide, \cite{NICE}).

The axiomatic assumptions over patient survival (Assumptions \ref{ass:surv}-\ref{ass:converging}) enable us to reasonably restrict the way survival data behave after the censoring time in a systematic way, yet leave us with some freedom and choice. At the time of censoring, based on the assumptions, there is a non-negative gap between hazard rates of $C$ and $A$. It remains to specify how $\hrate{C}{t}$ and $\hrate{A}{t}$ might increase and also how their difference might decline after the censoring time. Having the hazard rate functions enables us to complete the survival table, and from there, we can calculate the critical disutility.

The two extremes in extrapolating survival curves, which as we will prove in the next theorem will result in absolute upper and lower bounds for the critical disutility, are assuming an exponential distribution (with the last observed hazard rate as the parameter), and the case of a patient's survival dropping to zero (also known as the `stop and drop' assumption) for the post-censoring time. Although none of these assumptions seem to be suitable in reality, they provide bounds on the optimal policy's critical factor.

To get tighter bounds, in the following theorem we prove bounds for the critical disutility when hazard rates of random variables $A$ and $C$ are assumed to be constrained in pre-specified conic regions. These conic regions can, for instance, correspond to a case in which derivatives of hazard rate functions are assumed to be bounded in the post-censoring period. Empirically, these bounds on the derivatives can be found using pre-censoring data. For example, we can calculate the maximum unit time increment of the hazard rates in the existing data and consider that as an upper bound on the derivative of the hazard rate function.


\begin{thm}\label{thm:censor} Consider that a patient's access-based survival table is (right) censored at time $t'$. In addition to Assumptions \ref{ass:dec}-\ref{ass:qol}, assume that random variables $A$ and $C$ have hazard rate functions that are constrained beyond $t'$ in two respective cones characterized by parameters $(\alpha_A, \beta_A)$  and $(\alpha_C, \beta_C)$ described below (see Figure \ref{fig:cones}):
\begin{align*}
\alpha_A \le \frac{\hrate{A}{s}-\hrate{A}{t'}}{s-t'}  \le \beta_A: s >t', \\
\alpha_C \le \frac{\hrate{C}{s}-\hrate{C}{t'}}{s-t'}  \le \beta_C: s > t'.
\end{align*}
To have a non-void feasible set, it is necessary to assume $\alpha_A \le \beta_C$ and $\alpha_C \le \beta_A$. Assuming so, for all $t$ such that $t+M \le t'$ (with probability 1) we get
\begin{align*}
d^{cr}_l(t) \le d^{cr}(t) \le d^{cr}_u(t),
\end{align*}
where the lower and the upper bounds are achieved by assuming hazard rate functions $\mathbf{r}^{l}_A(s)$, $\mathbf{r}^{l}_C(s)$ and $\mathbf{r}^{u}_A(s)$, $\mathbf{r}^{u}_C(s)$, respectively defined below for $s>t'$:\\

\begin{tabular}{ll}
~$
\begin{cases}
\mathbf{r}^{u}_A(s)=\mathbf{r}_A(t') +\max \{\alpha_A, \alpha_C \}(s-t') \\
\mathbf{r}^{u}_C(s)=\mathbf{r}_C(t') +\max \{\alpha_A, \alpha_C \}(s-t')
\end{cases}
$;
&~$
\begin{cases}
\mathbf{r}^{l}_A(s)=\min \{ \mathbf{r}_A(t') + \beta _A (s-t') , \mathbf{r}_C(t') + \beta _C (s-t') \} \\  
\mathbf{r}^{l}_C(s)= \max \{ \mathbf{r}_C(t') + \alpha_C (s-t'), r_A^l(s) \}
\end{cases}$

\end{tabular}
\end{thm}


As the theorem states, these bounds are valid for values of $t$ that are far enough from the censoring time ($t+M \le t'$ with probability 1). This condition assures that at the time of censoring, $t'$, the AVF maturation period is over.

\begin{figure}[h]
\centering
\includegraphics[scale=.70]{./files/cones.pdf}
\caption{Cones restricting the hazard rate function of random variables $A$ and $C$ in the unobserved period.}
\label{fig:cones}
\end{figure}

In the following theorem, we prove that by tightening the cones that restrict the hazard rate functions we get tighter bounds on the critical disutility.
\begin{thm}\label{thm:tighening} In Theorem \ref{thm:censor}, consider two set of cones indexed by 1 and 2 whose parameters satisfy the following assumptions:
\begin{enumerate}
\item $\alpha_A^2 \le \alpha_A^1$
\item $\alpha_C^2 \le \alpha_C^1$
\item $\beta_A^1 \le \beta_A^2$
\item $\beta_C^1 \le \beta_C^2$
\end{enumerate} 
Then, we have $d^{cr,2}_l(t) \le d^{cr,1}_l(t)$ and $d^{cr,1}_u(t) \le d^{cr,2}_u(t)$.
\end{thm}
Based on this theorem,  by assuming $\alpha_A=\alpha_C=0$ and $\beta_A=\beta_C=\infty$, we can prove that the two extrapolation extreme cases mentioned before result in the highest upper and smallest possible lower bounds.
\section{Numerical Results}\label{sec:num}
To demonstrate the results of Theorems \ref{thm:QALE} and \ref{thm:cdis}, we performed a numerical study. The baseline values for different parameters of the model and sources used are given in Table \ref{tab:params}. 
\input{sheet1.tex}

For patients on-HD survival, we used  \cite{perl_hemodialysis_2011} which provides only 1-5 year survival probability for a 67 year old patient. Since the existing survival data is censored, we use Theorem \ref{thm:censor} to find a lower bound and an upper bound for the critical disutility. We calculated the average increase rate of the AVF's and CVC's hazard rate functions and use them as proxies for $\beta_A=\beta_C$ , and $\alpha_A=\alpha_C$, respectively. Then, we calculated hazard rate functions that result in the lower and upper bounds for the critical disutility, and used the former as the baseline value as well (see Figures \ref{fig:lower} and \ref{fig:upper}). Based on these hazard rate functions, patient's survival was calculated for each case (see Figure \ref{fig:survivalLU}). We also considered the two mentioned extreme cases for extrapolation, which based on Theorem \ref{thm:tighening}, result in the absolute lower and upper bounds for the critical disutility. Figure \ref{fig:CriticDisuAll} shows the critical disutility under different extrapolation scenarios.

\begin{figure}[h!]
  \centering
  \subfloat[Hazard rate functions for the lower bound case]{\label{fig:lower}\includegraphics[width=0.5\textwidth]{./files/lower.pdf}}
  \subfloat[Hazard rate functions for the upper bound case]{\label{fig:upper}\includegraphics[width=0.5\textwidth]{./files/upper.pdf}}
  \\
    \subfloat[Access-based survival probability for the lower and upper bound cases.]{\label{fig:survivalLU}\includegraphics[width=0.5\textwidth]{./files/projected.pdf}}
    \subfloat[Critical disutility under different scenarios]{\label{fig:CriticDisuAll}\includegraphics[width=0.5\textwidth]{./files/cdisutilALL.pdf}}
\caption{On-HD survival probability, hazard rate functions, and critical disutility for a 67 year old patient under different scenarios.}
\end{figure}
For each individual, the critical age can be found using the critical disutility plot (replotted for illustration in Figure \ref{fig:cdis67} under the baseline assumption for survival extrapolation); for instance the critical on-HD durations for patients with AVF creation disutility of 85 and 65 QALE days are 2 and 3 years, respectively.

\begin{figure}[h!]
  \centering
  \subfloat[Hazard rate functions for the lower bound case]{\label{fig:cdis67}\includegraphics[width=0.5\textwidth]{./files/cdis67.pdf}}
  \subfloat[Hazard rate functions for the upper bound case]{\label{fig:cdis82}\includegraphics[width=0.5\textwidth]{./files/cdis82.pdf}}
  
\caption{On-HD survival probability, hazard rate functions, and critical disutility for a 67 year old patient under different scenarios}
\end{figure}



Studies show that patients starting HD at higher ages have higher annual failure rate on either accesses (see \cite{KurellaSurv, Astor, CARVALHO}). Also, the difference in failure rate between HD on an AVF and a CVC will decrease with age of HD onset (see \cite{Astor}). Older patients also have a higher chance for AVF failure (\cite{Peterson,Levin}). Therefore, based on Theorem \ref{thm:compdcrt}, we can deduce that a patient starting HD at higher ages has a smaller critical disutility.

To see the impact of age of HD onset on the critical disutility, in Figure \ref{fig:cdis82}, we have plotted the critical disutility curves for patients who start HD at the age of 67 and 82 years. As the plot shows, the critical disutility of the older patient is always smaller. As a result, for a fixed AVF creation disutility, the patient who start at age 67 will have a higher critical on-HD duration; for instance for a disutility of 55 QALE days, the critical on-HD duration is 1 and 3.5 years for the 82 and 67 year old patients, respectively.

We also performed a sensitivity analysis to see how robust the results are to changes in the input parameters. The parameters and values tested for one-way and two-way sensitivity analyses and the corresponding critical disutilities are given in Table \ref{tab:SA}.
% ---------------------
\begin{table}[htbp]
  \centering
  \caption{Sensitivity analysis for the critical disutility (QALE days) of a 67 year old patient. The default values for each parameter is given in Table \ref{tab:params}.}
      \label{tab:SA}%
    \begin{tabular}{p{4cm}  l  c c c c c}
    \toprule
	\multirow{2}{*}{Parameter} &\multirow{2}{*}{Value} & \multicolumn{5}{c}{On-HD duration (years)}\\ \cline{3-7}
	& & 1 &2& 3 & 4&5\\ \toprule 
	    N/A   & Default & 151   & 129   & 113   & 96    & 92 \\
	    \hline
    \multirow{2}[4]{*}{\parbox{4cm}{AVF Surgery Success Probability}} & 0.2   & 76    & 65    & 57    & 48    & 46 \\
          & 0.6   & 223   & 189   & 165   & 140   & 133 \\\hline
    \multirow{2}[4]{*}{\parbox{4cm}{Functional AVF Annual Failure Rate}} & 0.1   & 172   & 147   & 130   & 112   & 108 \\
          & 0.2   & 134   & 114   & 100   & 84    & 80 \\\hline
    \multirow{3}[6]{*}{Maturation Time (months)} & Uniform [3,5] & 150   & 127   & 112   & 96    & 91 \\
          & Uniform [4,6] & 149   & 127   & 112   & 94    & 89 \\
          & Uniform [1,6] & 150   & 128   & 113   & 95    & 91 \\\hline
    \multirow{3}[6]{*}{QALE Coeff [CVC, AVF]} & [0.73,0.81] & 164   & 141   & 126   & 109   & 105 \\
          & [0.75,0.81] & 158   & 135   & 119   & 103   & 98 \\
          & [0.81,0.81] & 139   & 116   & 100   & 83    & 79 \\
    \bottomrule

    \end{tabular}%
\end{table}
% ---------------------










\section{Conclusion}\label{sec:dis}
Vascular access is the lifeline of HD patients. Survival and quality of life of patients on hemodialysis depends on the type of vascular access being used. AVFs, among other vascular access types, are associated with the best quality of life, highest longevity, lowest complication rates and lowest cost to the health care system in comparison with CVCs and AVGs. The importance of AVF in the care of ESRD patients is underscored by the Fistula First initiative.

Although AVFs are considered as the gold standard for hemodialysis, there are some limitations in terms of AVF use: 1. a vast majority of created AVFs does not mature to a point it can be used for HD 2. a mature and functional AVF has limited lifetime 3. patients have limited AVF opportunities 4. an AVF is created via a surgical procedure with its own inconveniences and complications.

In this work, we considered the problem of vascular access choice for HD patients. We assumed that patients can choose between an AVF and a CVC, provided that they have remaining AVF chances. We addressed the question of whether and when a patient may use an AVF opportunities and considered two metrics when comparing different policies: a patient's total lifetime, and QALE.

We analytically proved that delaying AVF referral stochastically decreases a patient's lifetime. As a result,  the policy of ``use the next AVF (opportunity) as soon as the one being used fails'' maximizes a patient's survival probability. The importance of this finding is that, not only does this policy maximize a patient's life expectancy, but it also maximizes the probability that the patient survives until a further time when she may switch to another treatment option, e.g. kidney transplantation.

By taking into account the inconvenience of AVF creation surgery, we defined a patient's QALE as the patient's quality adjusted lifetime on each access minus the AVF surgery disutility (for each surgeries performed). We proved that the optimal policy to maximize a patient's QALE is of a threshold type: there is an age threshold before which AVF is the optimal choice and we should follow the immediate AVF referral policy when an AVF fails (similar to the total lifetime policy), while after that age, CVC is the optimal vascular access choice. For instance, assuming 65 QALE days for AVF creation disutility, for a 67 year old patient, AVF surgery is the optimal action until the third year of HD.

Since AVF creation disutility is hard to estimate, we defined a dual view of the age-based threshold. We proved that at any time, the decision of whether to do an AVF surgery or not is determined by comparing the patient's AVF creation disutility with a critical value. In other words, the nephrologist can inform the patient on the benefit of undergoing the AVF creation surgery (i.e. the prospective additional quality lifetime), and the inconvenience of AVF creation surgery, and then the patient and the nephrologist can collectively decide whether to do the surgery or not. Thus, this facilitates nephrologists getting patients involved in the AVF referral decision making process, one of the key recommendations of the Institute of Medicine (\cite{IM}). This also addresses the considerable heterogeneity existing in treatment preference among ESRD patients, especially in the elderly (\cite{KurellaOpt, Vac}).

By assuming an exponential distribution for a patient's survival, we found a closed form for the AVF creation disutility which is also an upper-bound on the actual value. Note that the exponential distribution has been considered in the related research works (\cite{Xue}), and is a fair assumption when the patient has a short life expectancy.


We also found conditions under which we can compare the optimal policy for different patient types, e.g. based on this result and estimates in the literature, we proved that younger patients have a higher on-HD duration threshold.

The problem of vascular access choice in the elderly is a challenge receiving increasing attention (\cite{Vac}, \cite{KurellaFunc}). For instance, some authors have suggested that an attempt to place a permanent access should always be made in patients with at least 6 months of life expectancy (\cite{Vac, KurellaOpt}). Although the life expectancy plays an important role, they have not considered other key components such as AVF maturation rate and AVF creation disutility.


A few assumptions pose limitation to our study. We chose not to include AVGs as an alternative vascular access, since the prevalence of AVGs in HD patients has dramatically decreased in the past decade (it has decrease from 65\% to 20\% from 1995 to 2010; see \cite{Vassalotti, Kinney}). Also, AVGs are considered by US nephrologists as an alternative to an AVF primarily for the elderly or those who have a low chance of developing a mature AVF (e.g. those with a history of failed AVFs), whereas Canadian nephrologists prefer CVCs in these situations (\cite{Xi}).



\newpage
\section{Appendix I: Supplementary Results} \label{sec:AppSupp}
\begin{lem}\label{lem:hr_eq}
The followings are equivalent to $X \le_{hr} Y$:
\begin{enumerate}[(a)]
\item $X_t \le_{st} Y_t, \forall t.$
\item  $\dfrac{\surv{X}{t}}{\surv{Y}{t}} \downarrow t.$
\end{enumerate}

\end{lem}
\begin{proof}
~\\
For (a) see Theorem 1.B.7 in \cite{shaked2007stochastic}. For (b), see Theorem 1.3.3 in \cite{muller}.
\end{proof}
\begin{lem}[Closure of stochastic order under mixture]\label{lem:pres}
Let $X$, $Y$, $Z$ be random variables such that for all values of $z$, we have $[X | Z=z] \le_{st} [Y|Z=z]$. Then, $X \le_{st} Y$.
\end{lem}

\begin{proof}
See Theorem 1.2.15 in \cite{muller}.
\end{proof}

\begin{lem}\label{lem:ass2_a} Assumption \ref{ass:converging} is equivalent to having that $\dfrac{\surv{C}{t}}{\surv{A}{t}}$ is a log-convex function of $t$.
\end{lem}
\begin{proof}
Note that $\diff{\ln \surv{X}{t}}{t}=-\hrate{X}{t}$. Since $\diff{\ln \dfrac{\surv{C}{t}}{\surv{A}{t}}}{t}=\diff{\ln \surv{C}{t}}{t}-\diff{\ln \surv{C}{t}}{t}=\hrate{A}{t}-\hrate{C}{t}$, the result follows from Assumption \ref{ass:converging} and the fact that a differentiable function is convex if and only if its derivative is increasing.
\end{proof}
\begin{lem}\label{lem:log-conv}
Assume that function $f$ is differentiable and log-convex. Then $\frac{f(x)}{f(x+a)}$ is decreasing in $x$ for any $a \ge 0$.
\end{lem}
\begin{proof}
It suffices to show that $\ln\frac{f(x)}{f(x+a)}=\ln f(x)-\ln f(x+a)$ is decreasing in $x$. Define $F:=\ln f$, a convex function by assumption. Since $\diff{\ln\frac{f(x)}{f(x+a)}}{x}=\diff{F(x)}{x}-\diff{F(x+a)}{x} \le 0$, based on the fact that the derivative of a convex function is increasing, we have that $\ln\frac{f(x)}{f(x+a)}$ is decreasing in $x$.
\end{proof}
\begin{lem}\label{lem:IFR}
The random variable $X$ has the IFR property if and only if $X_t$ is stochastically decreasing in $t$.
\end{lem}
\begin{proof}
~\\
$\rightarrow$ Choose $t \le t'$ arbitrarily. Fix $s \ge 0$. We have $\hrate{X_{t'}}{s}=\hrate{X}{t'+s}$, and $\hrate{X_t}{s}=\hrate{X}{t+s}$. Thus, we have $\forall s, \hrate{X_t}{s} \le \hrate{X_t'}{s}$. Thus, $X_t' \le_{hr} X_t$, which implies $X_t' \le_{st} X_t$ by Lemma \ref{lem:hr_eq}.\\
$\leftarrow$ Choose $t \le t'$ arbitrarily. For all $s \ge 0$, we have $X_{t'+s} \le_{st} X_{t+s}$, thus by Lemma \ref{lem:hr_eq}, $X_{t'} \le_{hr} X_t$, and thus the result. 

\end{proof}
\begin{lem}\label{lem:res'} The mean residual lifetime of a random variable $X$ is differentiable, if $\surv{X}{t}$ is differentiable. Moreover, we have
$$\diff{\Ex X_t}{t}=\hrate{X}{t}\Ex X_t-1$$
\end{lem}
\begin{proof}
See \cite{gupta2003representing} for a proof.
\end{proof}

\newpage
\section{Appendix II: Results}
\subsection{Total Lifetime Results}
In this section, we show the proofs of the results in Section \ref{sec:optTL}.

The following lemma facilitates inferring results about the residual lifetime, and residual QALE by showing that at any time, the residual lifetime of random variables $A$, and $C$ also have the properties stated in Assumptions \ref{ass:relative}- \ref{ass:IFR}.
\begin{lem} \label{lem:assgen}
Assumptions \ref{ass:relative}-\ref{ass:IFR} apply to $A_t$, and $C_t$ as well. In other words, we have
$\forall t \ge 0$:
\begin{align*}
C_t \le_{hr} A_t,\\
\hrate{C_t}{s}-\hrate{A_t}{s} \downarrow s.
\end{align*}
\end{lem}

\begin{proof}
The result directly follows from the fact that $\hrate{X_t}{s}=\hrate{X}{t+s}$ for any random variable $X$, and $t,s\ge 0$
\end{proof}

Suppose one could set the AVF use time (rather than referral time) at $t=u$. We prove that the residual lifetime decreases stochastically in $u$. In Theorem \ref{thm:total}, we will show that this result extend to the case of AVF referral time. Before that, we provide further notation that will be used in what follows.
\begin{itemize}
\item $u$: time to use an AVF
\item $L(t,n)$: patient's residual lifetime at time $t$, given $n$ remaining AVF chances, under the optimal policy (one that maximizes patient's survival function, point-wise; if such policy exists)
\item $L(t,n,u)$: patient's residual lifetime at time $t$, when we use the first AVF chance at $t+u$, and follow the optimal policy for the subsequent $n-1$ AVF chances
\end{itemize}

\begin{prop}\label{prop:total}
Under Assumptions \ref{ass:dec}-\ref{ass:AVFs}, $L(t,n,u_2) \le_{st} L(t,n,u_1)$, whenever $u_1 \le u_2$.
\end{prop}

\begin{proof}
~\\
Let $L(u):=\big[L(t,n,u) \big |K_n=k]$. By closure of stochastic order under mixture (Lemma \ref{lem:pres}), it suffices to prove that for all $k$, $\surv{L(u)}{a}$ is decreasing in $u$ (for all $a$)  . We prove this by induction on $n$.\\

\noindent $\rightarrow$ Base case: $n=1$: \\
Depending on the values of $u, a, k$ we can calculate $\surv{L(u)}{a}$ as follows. (See Figure \ref{fig:fig1}):
\begin{figure}[htbp]
\centering
\includegraphics[scale=0.7]{./files/LT-F1.pdf}
\caption{Possible cases for $\surv{L(u)}{a}$}
\label{fig:fig1}
\end{figure}
\begin{itemize}
\item Case 1: $a \le u$: We have $\surv{L(u)}{a} \xlongequal{A(\ref{ass:surv})} \pr [C_t>a]=\surv{C_t}{a}$.
\item Case 2: $[a-k]^+ \le u \le a$: We have
\begin{align*}
\surv{L(u)}{a}&=\pr [C_t>u , A_{t+u}>a-u  ]=\pr [C_t>u]\pr[A_{t+u}>a-u \big |C_t>u]\\
& \xlongequal{A(\ref{ass:surv})} \pr [C_t>a]\pr[A_{t+u}>a-u] \xlongequal{A(\ref{ass:surv})}\pr [C_t>u].\pr[A_t>a|A_t>u]=\surv{C_t}{u}\frac{\surv{A_t}{a}}{\surv{A_t}{u}}
\end{align*}
\item Case 3: $0 \le u \le [a-k]^+$: We have:
\begin{align*}
\surv{L(u)}{a}&=\pr [C_t>u , A_{t+u}>a-u  , C_{t+u+k}>a-(u+k) ]\\
&=\pr [C_t>u].\pr[A_{u+t}>k \big |C_t>u].\pr[C_{u+k}>a-(u+k) \big |A_{t+u}>k , C_t>u]\\
&\xlongequal{A(\ref{ass:surv})} \pr [C_t>u].\pr[A_{t}>k+u \big |A_t>u].\pr[C_{t}>a \big |C_{t}>u+k]\\
& \xlongequal{A(\ref{ass:surv})} \surv{C_t}{u}.\frac{\surv{A_t}{k+u}}{\surv{A_t}{u}}\frac{\surv{C_t}{a}}{\surv{C_t}{u+k}}=
\surv{C_t}{a}.\frac{\surv{C_t}{u}}{\surv{A_t}{u}}/\frac{\surv{C_t}{u+k}}{\surv{A_t}{u+k}}
\end{align*}
\end{itemize}
in which $A(n)$ represents implication of Assumption $n$.
Note that $\surv{L(u)}{a}$ is continuous within each range, and its value on the boundary points coincides. Therefore, it suffices to prove that in each range, $\surv{L(u)}{a}$ is decreasing. In Case 1, the function is constant and thus the result holds trivially. In Case 2, since $C_t \le_{hr} A_t$ according to Lemma \ref{lem:assgen}, the function is decreasing using Lemma \ref{lem:hr_eq}. In Case 3, Lemma \ref{lem:assgen} and Lemma \ref{lem:ass2_a} imply that $\frac{\surv{C_t}{u}}{\surv{A_t}{u}}$ is log-convex in $u$. Using Lemma \ref{lem:log-conv}, we have that $\surv{L(u)}{a}$ is decreasing in $u$.\\


\noindent $\rightarrow$ Induction step: Assume $L(t,n-1,u_2) \le_{st} L(t,n-1,u_1)$, for all $u_1 \le u_2$. We prove that if $u_1 \le u_2$, then $L(t,n,u_2) \le_{st} L(t,n,u_1)$.\\

Based on the fact that stochastic order is a partial order, we can instead prove that $L(u_2) \le_{st} L'$ and $L' \le_{st} L(u_1)$, in which $L'$ is the lifetime under a hypothetical situation similar to $L(u_1)$ with the difference that the decision to use the subsequent AVF is delayed until $u_2+k$ (See Figure \ref{fig:prop1-induction}).
\begin{figure}[htbp]
\centering
\includegraphics[scale=0.70]{./files/LT-F2.pdf}

\caption{Induction step and the hypothetical random variable $L'$}
\label{fig:prop1-induction}
\end{figure}
\begin{itemize}
	\item $L(u_2) \le_{st} L'$: For $x \le u_2+k$, we have that $\surv{L(u_2)}{x}=\surv{L(t,1,u_2)}{x}$, and $\surv{L'}{x}=\surv{L(t,1,u_1)}{x}$. Thus the result follows from induction base. Otherwise, we have 
	\begin{align*}
		\surv{L(u_2)}{x}=\surv{L(u_2)}{u_2+k}.& \surv{L(u_2+k,n-1)}{x-[u_2+k]}, \\
		\surv{L'}{x}=\surv{L'}{u_2+k}.& \surv{L(u_2+k,n-1)}{x-[u_2+k]}.
	\end{align*}

	Based on the previous result, we have $\surv{L(u_2)}{u_2+k} \le \surv{L'}{u_2+k}$, and thus we get the result.
	\item $L' \le_{st} L(t,n,u_1)$. For $x \le u_1+k$, we have that $\surv{L(u_1)}{x}=\surv{L'}{x}=\surv{L(t,1,u_1)}{x}$. For $x  \ge u_1+k$, 
		\begin{align*}
			\surv{L(u_1)}{x}=\surv{L'}{u_1+k}.& \surv{L(u_1+k,n-1,0)}{x-[u_1+k]}, \\
			\surv{L'}{x}=\surv{L'}{u_1+k}.& \surv{L(u_1+k,n-1,u_2-u_1)}{x-[u_1+k]}.
		\end{align*}
		using the induction hypothesis, $L(u_1+k,n-1,u_2-u_1) \le_{st} L(u_1+k,n-1,0)$, we get the desired result.
\end{itemize}
\end{proof}

Now we extend the result of Proposition \ref{prop:total} from AVF use time to AVF referral time.\\

\begin{reptheorem}{thm:total}
Under Assumptions \ref{ass:dec}-\ref{ass:AVFs}, delaying AVF referral stochastically decreases a patient's lifetime.
\end{reptheorem}

\begin{proof}
~\\
In Proposition \ref{prop:total}, we proved that the patient's residual lifetime stochastically decreases in AVF use time. Since later referral means later AVF use time, then the patient's lifetime is stochastically decreasing in the referral time, as well. Mathematically, we can prove it as follows: let $r$ be the referral time, and $L_r$ the lifetime of the patient when the patient is referred at $r$ for the current AVF, and optimally (with respect to survival function) for the subsequent chances. Note that for AVF use time we have $u=r+W+M$. Fix $W=w,M=m$, arbitrarily. If $r_1\le r_2$, then $u_1 \le u_2$. As a result of Proposition \ref{prop:total}, 
$$\forall w,m:[L_{r_2}\big| W=w, M=m] \le_{st} [L_{r_1}\big|W=w, M=m].$$ 
Now, by closure of stochastic order under mixture (Lemma \ref{lem:pres}), this implies $L_{r_2} \le_{st} L_{r_1}$.
\end{proof}

\subsection{QALE -- A Generic Patient}
In this section, we provide the proofs for Section \ref{sec:QALEGP}.

We restrict ourselves to threshold policies that are not dominated by $\pi_0$ when applied at any decision state $(t,n)$. Let $T$ be the set of such thresholds, i.e. $$T=\{\tau: v^{\pi(\tau)}(t,n) \ge {v^{\pi_0}(t,n)} : \forall t,n  \}.$$
Note that $T$ is non-empty since $0 \in T$. We will show that there exist $ \tau^* \in T$ such that $\pi(\tau^*)$ is optimal. In order to prove this, we first show that for all $\tau \in T$, the difference of the performance of the threshold policy $\pi(\tau)$ and $\pi_0$ decreases in $t$. We prove some preliminary results before proving this claim in Proposition \ref{prop:thresh_dec}.

\begin{lem} \label{lem:dec_v}
$ \diff{v(t,1,0|M_1=m,K_1=k)}{k}$ is non-negative and decreasing in $t$.
\end{lem}
\begin{proof}
~\\
Let $w(t,m,k):=v(t,1,0|M_1=m,K_1=k)$. We can calculate $w(t,m,k)$ as follows:
\begin{align} \label{eq:decind} \nonumber
w(t,m,k)=-d+ q_c\int_{0}^{m} x\pdf{C_t}{x}dx\\
 +\surv{C_t}{m}\bigg [q_cm+ & q_a\int_{0}^{k} x \pdf{A_{t+m}}{x} dx+\surv{A_{t+m}}{k} \big[q_ak+q_c\Ex C_{t+m+k} \big ] \bigg].
\end{align}
We can establish differentiability of $w$ as follows. Note that $\forall x\ge 0$,  $\surv{A_{x}}{k}$ and $\surv{C_{x}}{k}$ are differentiable in $k$ by Assumption \ref{ass:surv}. Differentiability pf the latter itself implies differentiability of  $\Ex C_{x+k}$ by Lemma \ref{lem:res'}. These two imply that $w(t,m,k)$ is differentiable in $k$.\\

Using Lemma \ref{lem:res'}, we have
\begin{align} \label{eq:w'}
 \nonumber \diff {w(t,m,k)}{k}=& \surv{C_t}{m} \bigg [ \diff{q_a\int_{0}^{k} x \pdf{A_{t+m}}{x} dx}{k} + \surv{A_{t+m}}{k} \diff{\big[q_ak+q_c\Ex C_{t+m+k} \big ]}{k} +  \big [\diff{\surv{A_{t+m}}{k}}{k}\big ]\big[q_ak+q_c\Ex C_{t+m+k} \big ] \bigg ]\\ \nonumber
=& \surv{C_t}{m} \bigg [ q_ak\pdf{A_{t+m}}{k}+\surv{A_{t+m}}{k} \big\{ q_a+q_c \big [\hrate{C_{t+m}}{k} \Ex C_{t+m+k}-1]\big\} 
-\pdf{A_{t+m}}{k}  [q_ak+q_c \Ex C_{t+m+k} ]  \bigg] \\
=& \surv{C_t}{m} \surv{A_{t+m}}{k} \bigg[q_a-q_c+q_c \Ex C_{t+m+k} \big [\hrate{C_{t}}{m+k}-\hrate{A_t}{m+k}\big] \bigg]
\end{align}

We can prove that $\diff {w(t,m,k)}{k}$ is decreasing in $t$, and non-negative by showing that it is product of the following three non-negative decreasing functions:

\begin{enumerate}
\item $\surv{C_t}{m}$: This is decreasing in $t$, since $C_t$ is stochastically decreasing in $t$ based on Assumption \ref{ass:IFR} and Lemma \ref{lem:IFR}.
\item $\surv{A_{t+m}}{k}$: This is decreasing in $t$, since $A_{t+m}$ is stochastically decreasing in $t$ based on Assumption \ref{ass:IFR} and Lemma \ref{lem:IFR}.
\item $q_a-q_c+q_c \Ex C_{t+m+k} \big [\hrate{C_{t}}{m+k}-\hrate{A_t}{m+k}\big]$:
\begin{itemize}
\item non-negative: We have that $q_a \ge q_c$. Also, $\hrate{C_{t}}{m+k} \ge \hrate{A_t}{m+k}$ based on Lemma \ref{lem:assgen}.
\item decreasing: $\Ex C_{t+m+k}$ is decreasing in $t$, because $C_{t+m+k}$ is stochastically decreasing in $t$. Also, $\hrate{C_{t}}{m+k}-\hrate{A_t}{m+k}$ is decreasing in $t$ based on Lemma \ref{lem:assgen}.
\end{itemize} 
\end{enumerate}
\end{proof}
The following intermediate results will provide a foundation for validity of the binary search for the optimal threshold in later results:
\begin{prop} \label{prop:v1}
$ v(t,1,0)-v^{\pi_0}(t,1)$ is decreasing in $t$.
\end{prop}
\begin{proof}
Choose $t_1 \le t_2$ arbitrarily. We have that $\forall m:\diff {[w(t_2,m,k)-w(t_1,m,k)]}{k} \le 0$ by the linearity of the differential operator, and Lemma \ref{lem:dec_v}. This implies that $$\forall k,m: w(t_2,m,k)-w(t_1,m,k) \le w(t_2,m,0)-w(t_1,m,0).$$
But, $\forall m:w(t,m,0)=-d+v^{\pi_0}(t,1) $. Thus, 
$$\forall k,m:  w(t_2,m,k)-w(t_1,m,k) \le v^{\pi_0}(t_2,1)-v^{\pi_0}(t_1,1).$$
Since $v(t,1,0) = \Ex_{M_1,K_1} w(t,m,k)$ by Assumption \ref{ass:AVFs}, taking expectation from both sides with respect to $M_1,K_1$ gives us:
$$v(t_2,1,0)-v(t_1,1,0) \le v^{\pi_0}(t_2,1)-v^{\pi_0}(t_1,1).$$
\end{proof}
Note that  if any of Assumptions \ref{ass:relative}-\ref{ass:IFR} hold strictly, we have that the monotonicity is strict for Lemma \ref{lem:dec_v}, and Propositions \ref{prop:v1} and \ref{prop:thresh_dec}.

Now, we prove our claim:
\begin{prop} \label{prop:thresh_dec} For all $\tau \in T$, we have $\forall n: v^{\pi(\tau)}(t,n)- {v^{\pi_0}(t,.)} \downarrow t$.
\end{prop} 
\begin{proof}
~\\ Note that since $M_i,K_i$ are independent of the survival process and the policy in use (based on Assumption \ref{ass:AVFs}), we have that 
\begin{align*}
v^{\pi}(t,n)=\Ex \big [ v^{\pi}(t,n \big |M_1,\ldots, M_n,K_1,\ldots, K_n)\big].
\end{align*}
We prove the result by induction on $n$ as follows:
\begin{itemize}
\item $n=1$: For $v^{\pi(\tau)}(t,1)$ we have:
\begin{align*}
v^{\pi(\tau)}(t,1)- v^{\pi_0}(t,1)=
\begin{cases}
v(t,1,0)-v^{\pi_0}(t,1)& t \le \tau \\
0& o.w.
\end{cases}
\end{align*}

The function is decreasing  for $t \le \tau$ by Corollary \ref{prop:v1}, and for $t > \tau$ trivially. It suffices to have that $v^{\pi(\tau)}(\tau,n) - v^{\pi_0}(\tau,1) \ge 0$, which holds by the fact that $\tau \in T$.
\item Assume the result holds for $n=1,\ldots, k$. We prove that it holds for $n=k+1$.

To that end, we first prove the following:
\begin{align} \label{eq:relative}
\forall n>1: v^{\pi(\tau)}(t,n) - v^{\pi(\tau)}(t,n-1) \downarrow {t}.
\end{align}
Since $v^{\pi(\tau)}(t,k)- {v^{\pi_0}(t,.)}=[v^{\pi(\tau)}(t,k+1)- v^{\pi(\tau)}(t,k)]+[v^{\pi(\tau)}(t,k)- {v^{\pi_0}(t,.)}]$, the result will then follow from the fact that the sum of two decreasing functions is decreasing.\\
Now we show Equation \ref{eq:relative} as follows:\\
We first fix $M_i=m_i$, and $K_i=k_i$ for $i=2,\ldots,n$ arbitrarily.  The result generalizes using the preservation of monotonicity under expectation.\\
Let $t':=\sum_{i=2}^{n} m_i+k_i$, and based on whether $ t+t' \le \tau$, consider two cases:
\begin{itemize}
\item $t+t' \le \tau$: In this case the last referral happens at time $t+t'$, provided that patient survives until that time; The difference in the cases of $n$, and $n-1$ remaining AVF chances is the (possible) use of one AVF chance. Let $S(t,t')$ represent the probability of survival of the patient until time $t'$. Then, we have the following:
\begin{align} \label{eq:deltav}
[ v^{\pi(\tau)}(t,n)- v^{\pi(\tau)}(t,n-1)]\big | M_{2,\ldots,n},K_{2,\ldots,n} =S(t',t) \big[v^{\pi(\tau)}(t+t',1)-v^{\pi_0}(t+t',1)\big].
\end{align}
\item Otherwise; $\exists j: t+\sum_{i=j+1}^{n} [m_i+k_i] > \tau$. Based on $\pi(\tau)$, we don't refer for the remaining AVF chances after $\tau$, and thus we have  
$v^{\pi(\tau)}(t,n|M_{2,\ldots,n},K_{2,\ldots,n})=v^{\pi(\tau)}(t,n-l|M_{2,\ldots,n},K_{2,\ldots,n})$ for all $l \le j$. \\
\end{itemize}
Thus, we have
\begin{align}\label{eq:deltavn}
[ v^{\pi(\tau)}(t,n)- v^{\pi(\tau)}(t,n-1)]\big | M_{2,\ldots,n},K_{2,\ldots,n}=
\begin{cases}
S(t',t) \big[v^{\pi(\tau)}(t+t',1)-v^{\pi_0}(t+t',1)\big]&:  t +t' \le  \tau \\
0 &: o.w.
\end{cases}
\end{align}

It suffices to show that $S(t',t) \big[v^{\pi(\tau)}(t+t',1)-v^{\pi_0}(t+t',1)\big] $ is decreasing in $t$ and non-negative. We prove that by showing that it is the product of the following two non-negative and decreasing functions:
\begin{itemize} 
\item $S(t',t)$: The probability is non-negative by definition. First we compute $S(t',t)$ as follows:
\begin{align*}
S(t',t)=&\pr [C_t>m_n , A_{t+m_n}>k_n,\ldots ,A_{t+t'-k_2}>k_2]\\
=&\pr[C_t>m_n] \pr[A_{t+m_n}>k_n|C_t>m_n]\ldots  \pr [A_{t+t'-k_2}>k_2|C_t>m_n ,\ldots]\\
&\xlongequal{A(\ref{ass:surv})}\surv{C_t}{m_n}\surv{A_{t+m_n}}{k_n}\ldots \surv {A_{t+t'-k_2}}{k_2}
\end{align*}
Each of the survival probabilities are decreasing in $t$ because $A_{t+x}$ and $C_{t+x}$ are stochastically decreasing in $t$, for any $x\ge 0$ by Assumption \ref{ass:IFR}.
\item $v^{\pi(\tau)}(t+t',1)-v^{\pi_0}(t+t',1)$: This term is non-negative since $\tau \in T$. Also, this term is decreasing in $t$ using the result of the case $n=1$.
\end{itemize}
\end{itemize}
\end{proof}

The following corollary is used in extending the optimality of the policy found for $n=1$ to $n>1$.
\begin{cor} \label{cor:NWN}
If $\forall t: v^{\pi(\tau)}(t,1) \ge v^{\pi_0}(t,1)$, then  $v^{\pi(\tau)}(t,n)$ is increasing in $n$. As a result  $\tau \in T$.
\end{cor}
\begin{proof}
~\\

Since $\forall t: v^{\pi(\tau)}(t,1) \ge v^{\pi_0}(t,1)$ by assumption, from Equation \ref{eq:deltavn}, we have $$[ v^{\pi(\tau)}(t,n)- v^{\pi(\tau)}(t,n-1)]\big | M_{2,\ldots,n},K_{2,\ldots,n}  \ge 0.$$ Taking expectation with respect to $ M_{2,\ldots,n},K_{2,\ldots,n}$, we get:
\begin{align*} 
 v^{\pi(\tau)}(t,n) \ge v^{\pi(\tau)}(t,n-1)\ge \ldots \ge v^{\pi(\tau)}(t,1) \ge v^{\pi_0}(t,1).
\end{align*}
\end{proof}
Before proving the main results, we prove the following preliminary result.
\begin{lem} \label{lem:delta}
The following equality holds for $v(t,n,y)$.
\begin{align*} 
v(t,n,y)&=\surv{C_t}{y}\bigg[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\bigg]+v^{\pi_0}(t,n)
\end{align*}
\end{lem}

\begin{proof}
 ~\\We can calculate $v(t,n,y)$ as follows:
\begin{align*}
v(t,n,y)&=q_c\int_{0}^{y} x\pdf{C_t}{x} dx +\surv{C_t}{y} \big [q_c y+v(t+y,n,0) \big ]
\end{align*}
After rearranging terms, we get:
\begin{align*}
v(t,n,y)&=\surv{C_t}{y}\big[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\big]+q_c\int_{0}^{y} x\pdf{C_t}{x} dx+\surv{C_t}{y} \big [q_c y+v^{\pi_0}(t+y,n) \big]\\
	&=\surv{C_t}{y}\big[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\big]+q_c\left[ \int_{0}^{y} x\pdf{C_t}{x} dx+\surv{C_t}{y} [ y+\Ex C_{t+y} ]\right]\\
	&=\surv{C_t}{y}\big[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\big]+q_c\Ex C_{t}\\
	&=\surv{C_t}{y}\big[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\big]+v^{\pi_0}(t,n).
\end{align*}
\begin{figure}[htbp]
\centering
\includegraphics[scale=0.6]{./files/QALE-F1.pdf}
\caption{$v(t,n,y)$ versus $v^{\pi_0}(t,n)$ }
\label{fig:lem}
\end{figure}
Consider Figure \ref{fig:lem}. Another way of proving this equality is by calculating $v(t,n,y)- v^{\pi_0}(t,n)$ as follows:
\begin{align*} 
v(t,n,y)- v^{\pi_0}(t,n)=&\bigg[\big(v(t,n,y)- v^{\pi_0}(t,n)\big) \big | C_t \le y \bigg]\pr [C_t \le y]+\bigg[\big(v(t,n,y)- v^{\pi_0}(t,n)\big) \big | C_t > y \bigg]\pr [C_t > y]\\
&=\pr [C_t \le y](0)+\pr [C_t > y]\big[v(t+y,n,0)- v^{\pi_0}(t+y,n)\big]\\
&=\surv{C_t}{y}\big[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\big].
\end{align*}
\end{proof}
\noat{I guess we can remove the first proof.\\}
The following theorem proves the optimality of threshold policies for the case we have one AVF opportunity ($n=1$), and also establishes a base for finding the optimal threshold.
\begin{reptheorem}{thm:qalen=1}[Existence of a referral threshold for $n=1$]
Assume $n=1$. There is an age threshold $\tau^*$ such that the policy $\pi(\tau^*)$ maximizes the expected QALE of the patient. In other words, for $t < \tau^*$, referral at $t$ is the optimal action, otherwise the no-referral action is  optimal.
\end{reptheorem}
\begin{proof}
~\\
Fix $t$, and $n$.  Assume that we take action $@_y$. By Lemma \ref{lem:delta}, we have:
\begin{align*}
v(t,1,y)=\surv{C_t}{y}\big[ v(t+y,n,0)-v^{\pi_0}(t+y,n)\big]+v^{\pi_0}(t,n)
\end{align*}
For $@_y$ to be an optimal action candidate, it is necessary that referral at $t+y$ is no worse than the no-referral action, i.e. $v(t+y,1,0) \ge v^{\pi_0}(t+y,1)$.\\
Since $v(t+y,1,0) - v^{\pi_0}(t+y,1)$ is decreasing in $y$ by Corollary \ref{prop:v1}, and $\surv{C_t}{y}$ is decreasing in $y$, then $v(t,1,y)$ is decreasing in $y$ for all $y$ that satisfy the necessary condition. Thus, the optimal action is to refer at $t$ if $v(t,1,0) \ge v^{\pi_0}(t,1)$, and no-referral, if otherwise.

Now, we form the policy $\pi(\tau^*)$ as follows based on whether $v(0,1,0) < v^{\pi_0}(0,1)$ or not.
\begin{itemize}
\item $v(0,1,0) < v^{\pi_0}(0,1)$: we have that for $\forall t:v(t,1,0) < v^{\pi_0}(t,1)$, since $v(t,1,0) - v^{\pi_0}(t,1)$ is decreasing in $t$ by Corollary \ref{prop:v1}. As a result, the no-referral action is optimal for all $t$. Choose  $\tau^*=0$ in this case.
\item $v(0,1,0) \ge v^{\pi_0}(0,1)$: we have that $\exists t'\le \infty$ such that for $t \le t'$, we have $v(t,1,0) \ge v^{\pi_0}(t,1)$, and $v(t,1,0) < v^{\pi_0}(t,1)$ otherwise. For $t \le t'$, referral at $t$ is optimal, and the no-referral action is optimal, if otherwise. Choose $\tau^*=t'$ in this case.
\end{itemize}
The policy $\pi(\tau^*)$ is optimal for $n=1$ by construction.
\end{proof}

\begin{repcorollary}{cor:binsearch}[Binary search] The optimal policy can be found using a binary search for $\tau^*$ over $[0,\infty)$.
\end{repcorollary}
\begin{proof} See how the optimal policy is formed in Theorem \ref{thm:qalen=1}.
\end{proof}
In the next theorem, we see that the optimal threshold found in Theorem \ref{thm:qalen=1} for $n=1$ is also optimal for $n > 1$.
\begin{reptheorem}{thm:QALE} [Optimality of threshold policies]
The policy $\pi(\tau^*)$ (constructed in Theorem \ref{thm:qalen=1}) is optimal for all $n \ge 1$.
\end{reptheorem}

\begin{proof}
~\\
We prove the theorem by induction on $n$:

\begin{itemize}
\item $n=1$: The policy $\pi(\tau^*)$ is optimal for $n=1$ by construction.
\item Assume for $n=1,\ldots, k$ the threshold policy $\pi(\tau^*)$ is optimal. We prove that, it is optimal for $n=k+1$ as well.\\
We prove the optimality of the policy $\pi(\tau^*)$ as follows:
\begin{itemize}
\item $t >\tau^*$: The policy suggests the no-referral action. We argue that it is optimal as follows.\\
Note that based on the optimality of $\pi(\tau^*)$ for $n \le k$, the last $k$ AVF chances won't be used, because their referral would happen at some $t' \ge t > \tau^*$.
Thus, we are left with one AVF chance. Similarly, we should not use that chance either. Thus, the no-referral action is optimal in this case.

\item  $t \le \tau^*$: The policy suggests referral at $t$. We argue that it is optimal as follows.\\
Note that no referral can be made later than $\tau^*$ (using the logic explained in the first case). Thus, we restrict our attention to $y \le \tau^*-t$. For all such $y$, we have that $v(t+y,n,0)= v^{\pi(\tau ^*)}(t+y,n)$, based on the induction assumption. By Lemma \ref{lem:delta}, we have
$$v(t,n,y)=\surv{C_t}{y}\bigg[v^{\pi(\tau ^*)}(t+y,n)-v^{\pi_0}(t+y,n)\bigg]+v^{\pi_0}(t,n).$$ 


 We conclude the proof by showing $v(t,n,y)$ is decreasing in $y$. Since $\surv{C_t}{y}$ is decreasing in $y$, it suffices to show that $v^{\pi(\tau ^*)}(t+y,n)-v^{\pi_0}(t+y,n)$ is non-negative and decreasing in $y$.

Since $\pi(\tau^*)$ is optimal for $n=1$, then $\forall t: v^{\pi(\tau^*)}(t,1) \ge v^{\pi_0}(t,1)$. As a result of Corollary \ref{cor:NWN}, we have $\tau^* \in T$ by . This implies  $$v^{\pi(\tau ^*)}(t+y,n) \ge v^{\pi_0}(t+y,n).$$ 

Since $\tau^* \in T$, using Proposition \ref{prop:thresh_dec}, we have that $v^{\pi(\tau ^*)}(t+y,n)- v^{\pi_0}(t+y,n)\downarrow y$.
\end{itemize}
\end{itemize}
\end{proof}

\subsection{Patient-Specific Considerations}
In this section, we provide the proofs of Section \ref{sec:psqale}.

We prove a preliminary result first:
\begin{thm}\label{thm:dec_d}
The critical age, $\tau^*(d)$, is decreasing in $d$.
\end{thm}
\begin{proof}
~
\\ Choose $0 \le d_1 \le d_2$ arbitrarily. By Theorem \ref{thm:QALE}, there exist $\tau^*_1$ and $\tau^*_2$ for which threshold policies $\pi(\tau^*_1)$ and $\pi(\tau^*_2)$ are optimal for cased $d=d_1$, and $d=d_2$, respectively.
Based on the proof of Theorem \ref{thm:qalen=1}, 
\begin{align} \label{eq:IFFTHLD}
t' \ge \tau^*(d) \iff v(t',1,0 ;d) < v^{\pi_0}(t',1)
\end{align}
As a result, $v(\tau^*_1,1,0 ;d_1) < v^{\pi_0}(\tau^*_1,1)$. But, $\forall t': v(t',1,0 ;d) - v^{\pi_0}(t',1)$ is (linearly) decreasing in $d$ (see Equation \ref{eq:decind}). Thus, $v(\tau^*_1,1,0 ;d_2) < v^{\pi_0}(\tau^*_1,1)$. By Equation \ref{eq:IFFTHLD}, we have $ \tau^*_2 \le \tau^*_1$.
\end{proof}


\begin{reptheorem}{thm:cdis} At any age, there exist a critical AVF creation disutility denoted by $d^{\text{cr}}(t)$, such that the optimal decision at time $t$ is to do AVF surgery immediately if patient's AVF creation disutility is not higher than the critical disutility (i.e. if $d \le d^{\text{cr}}(t)$), and is to use CVC for the rest of patient's life, otherwise.
\end{reptheorem}
\begin{proof}
~\\Fix $t$. Define $\Delta Q(t)$ as the benefit of an immediate AVF surgery vs using CVC for the rest of the patient's lifetime before subtracting the AVF creation disutility. In other words, $\Delta Q(t)=d+v(t,1,0) -v^{\pi_0}(t,1)$. By Theorem \ref{thm:d=0}, $\Delta Q(t) \ge 0$. By Proposition \ref{prop:thresh_dec}, we have that $\Delta Q(t)$ is decreasing in $t$. In Theorem \ref{thm:compdcrt}, we prove that $\Delta Q(t)$ is decreasing in $M$ and increasing in $K$. Thus, $\Delta Q(t) \le \Ex [A_t - C_t]$, in which the upper bound is achieved when $K=\infty$, and $M=0$ with probability one. Therefore, $\Delta Q(t)$ is finite. By Equation \ref{eq:IFFTHLD}, $t < \tau^*(d)$, and immediate surgery is optimal, if and only if $\Delta Q(t) \ge d$. Since $\Delta Q(t)$ is decreasing in $t$, we have $d^{\text{cr}}(t)=\Delta Q(t)$. 
\end{proof}
\begin{repcorollary}{cor:dcrt}
Critical disutility, $d^{\text{cr}}(t)$, is decreasing in $t$. Furthermore, if any of Assumptions \ref{ass:relative}-\ref{ass:IFR} hold strictly, we have that $d^{\text{cr}}(t)$ is strictly decreasing in $t$.
\end{repcorollary}
\begin{proof}
The results immediately follows from the definition of $\Delta Q(t)$ and Proposition \ref{prop:thresh_dec}.
\end{proof}

\begin{reptheorem}{thm:compdcrt}
Consider two patients types indexed by 1 and 2 whose characteristics hold the followings:
\begin{enumerate}
\item $M^{(2)} \le_{st} M^{(1)}$
\item $K^{(1)} \le_{st} K^{(2)}$
\item $A^{(1)} \le_{hr} A^{(2)}$, and $C^{(1)} \le_{hr} C^{(2)}$
\item $[\hrate{C^{(1)}}{t}- \hrate{A^{(1)}}{t}] \le [\hrate{C^{(2)}}{t} -\hrate{A^{(2)}}{t}]: \forall t$
\item $q_c^{(1)} \le q_c^{(2)}$, and $q_a^{(1)} - q_c^{(1)} \le q_a^{(2) }- q_c^{(2)}$
\end{enumerate}
where $(i)$ denotes the index. Then, $d^{\text{cr}}_{(1)}(t) \le d^{\text{cr}}_{(2)}(t): \forall t$.
\end{reptheorem}
\begin{proof}
~\\ Recall from Lemma \ref{lem:dec_v},  $v(t,1,0|M_1=m,K_1=k)$ is increasing in $k$. As a result, $\Delta Q(t|M=m, K=k)$ is increasing in $k$. Below, we show that it is decreasing in $m$. Similar to the proof of Lemma \ref{lem:dec_v}, we can show that $\diff{v(t,1,0|M_1=m,K_1=k)}{k}$ is decreasing in $m$. By following proof of Corollary \ref{prop:v1}, we have that
$$\forall k,t:  w(t,m_1,k)-w(t,m_2,k) \le v^{\pi_0}(t,1)-v^{\pi_0}(t,1)=0.$$
and hence $\Delta Q(t|M=m, K=k)$ is decreasing in $m$.


Using Equation \ref{eq:w'} and the assumptions 3-5 of the Theorem, we have that $\diff {w_{(1)}(t,m,k)}{k} \le \diff {w_{(2)}(t,m,k)}{k}$. By following proof of Corollary \ref{prop:v1}, we can show that
$$\forall k,t:  w_{(1)}(t,m,k)-w_{(2)}(t,m,k) \le v^{\pi_0}(t,1)-v^{\pi_0}(t,1).$$
and hence
$$\Delta Q_{(1)}(t|M=m, K=k) \le \Delta Q_{(2)}(t|M=m, K=k).$$
Using the fact that $\Delta Q(t|M=m, K=k)$ is increasing in $k$ and decreasing in $m$, and assumptions 1 and 2 of the Theorem, we have $\Delta Q_{(1)}(t) \le \Delta Q_{(2)}(t)$ and thus the result.
\end{proof}
\subsection{A Special Case}
\begin{reptheorem}{thm:exp}
Assume that $A \sim \exp(a)$, and $C \sim \exp(c)$, where $0<a \le c$. Then,
\begin{align}\label{eq:comp2}
d^{\text{cr}}(t) =d^{cr}:= p\bigg[\frac{q_a}{a}-\frac{q_c}{c} \bigg] \bigg[\Ex_M\big[ e^{-cM}\big]\bigg ] \bigg[1-\Ex_Z\big[ e^{-aZ}\big] \bigg].
\end{align}
where $p$ is the success probability of the AVF creation process, and $Z$ is the lifetime of a functional AVF. Furthermore, if $M\sim \exp(m)$, and $K\sim \exp(k)$, we have:
\begin{align*}
d^{\text{cr}} = p .\bigg[\frac{q_a}{a}-\frac{q_c}{c}\bigg] .  \frac{1}{1+ \frac{m}{c}} . \frac{1}{1+ \frac{a}{k}} 
\end{align*}
\end{reptheorem}
\begin{proof}
~\\Using equation \ref{eq:w'}, we have:
\begin{align*}
\diff {w(t,m,k)}{k}=e^{-cm}.e^{-ak}\big[q_a-q_c+q_c\frac{1}{c}(c-a)\big]=e^{-cm}.[ae^{-ak}]\big[\frac{q_a}{a}-\frac{q_c}{c}\big]
\end{align*}
Recall that $\forall m:w(t,m,0)=-d+v^{\pi_0}(t,1)$. Therefore, by taking integration with respect to $k$:
\begin{align*}
\Delta Q(t \big |M=m, K=k)=d+w(t,m,k)-v^{\pi_0}(t,1)=\big[\frac{q_a}{a}-\frac{q_c}{c} \big] e^{-cm}[1-e^{-ak}\big].
\end{align*}
Taking expectation with respect to $M$, and $K$ proves Equation \ref{eq:comp2}.

Now consider the case that $M$ and $K$ follow exponential distributions. Using the moment generation function of an exponential distribution $X \sim \exp(\lambda)$, we have
\begin{align}\label{eq:MGexp}
\Ex_X[e^{-tX}]=(1+\frac{t}{\lambda})^{-1}
\end{align}
Combining Equations \ref{eq:MGexp} and \ref{eq:comp2}, we get the desired result.
\end{proof}


\section{title}

\begin{reptheorem}{thm:pavf} Critical disutility is linear in AVF primary failure rate.
\end{reptheorem}

\begin{proof}
~\\ Let $Z$ be a Bernoulli random variable determining whether AVF creation is successful or not. Then, 
$$d^{\text{cr}}(t)=\Delta Q(t)=\Ex_Z \Delta Q(t|Z)=\pr[Z=1]\Ex \Delta Q(t|Z=1)+\Ex \Delta Q(t|Z=0)\pr[Z=0]=\pr[Z=1].\Ex Q(t|Z=1).$$
where the last equality uses the fact $\Ex \Delta Q(t|Z=0)=0$.
\end{proof}

\newpage
\bibliography{refs}
\newpage
\end{document}





In this section, we discuss how the optimal policy can ba tailored for each individual based on their personal preferences and/or their physical characteristics. Since we proved that the optimal policy is of threshold type for a generic patient, we compare the optimal threshold for patients with comparable characteristics.


\begin{cor}\label{cor:optTotal}
The optimal policy to maximize patient's survival probability after $x$ year(s) for all $x\ge 0$ (and as a result to maximize expected lifetime) is to refer a patient on CVC for an AVF as soon as possible, provided AVF opportunities remain and no AVF is already in the process of maturing.
\end{cor}

\begin{thm}\label{thm:censor2} Consider that a patient's access-based survival table is (right) censored at time $t'$. In addition to Assumptions \ref{ass:dec}-\ref{ass:qol}, assume that random variables $A$ and $C$ have hazard rate functions that are differentiable and have bounded derivatives beyond $t'$, i.e. assume $\delta_l \le \mathbf{r}'_A(t), \mathbf{r}'_C(t) \le \delta_u$ for all $t \ge t'$. Then, for all $t$ such that $t+M \le t'$ with probability 1, we have
\begin{align*}
d^{cr}_l(t) \le d^{cr}(t) \le d^{cr}_u(t),
\end{align*}
where the lower and the upper bound result from assuming hazard rate functions $\mathbf{r}^{l}_A(t)$, $\mathbf{r}^{l}_C(t)$ and $\mathbf{r}^{u}_A(t)$, $\mathbf{r}^{u}_C(t)$, respectively defined below:\\

\begin{tabular}{ll}
Upper bound: $\mathbf{r}^{u}_A(t)=\mathbf{r}_A(t')+\delta_l(t-t'): t\ge t' $; & $\mathbf{r}^u_C(t)=\mathbf{r}_C(t')+\delta_l(t-t'): t\ge t'$ \\ 
Lower bound:
$\mathbf{r}^{l}_A(t)=\mathbf{r}_A(t')+\delta_u(t-t'): t\ge t'$; & 
$\mathbf{r}^{l}_C(t)=
\begin{cases}
\mathbf{r}_C(t')+\delta_l(t-t'): & t' \le t \le t''\\
\mathbf{r}_A(t')+\delta_u(t-t'): & t > t''
\end{cases}$
\end{tabular}\\
in which $t''=\frac{\mathbf{r}_C(t')-\mathbf{r}_A(t')}{\delta_u - \delta_l}$.
\end{thm}